Add execution tracking and timeout monitoring for Jupyter cells #32
+672
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Track execution start/end with timestamps - Log execution duration and success status - Publish execution metadata to webapp - Register IPython event handlers for monitoring
- Monitor long-running executions with configurable thresholds - Send warnings when executions exceed warning threshold - Optional auto-interrupt for stuck executions via SIGINT - Report warnings/timeouts to webapp - Configurable via environment variables: - DEEPNOTE_ENABLE_EXECUTION_TIMEOUT - DEEPNOTE_EXECUTION_WARNING_THRESHOLD (default: 240s) - DEEPNOTE_EXECUTION_TIMEOUT_THRESHOLD (default: 300s) - DEEPNOTE_EXECUTION_AUTO_INTERRUPT (default: false)
- Add publish_execution_metadata() function - Define DEEPNOTE_EXECUTION_METADATA_MIME_TYPE constant - Publish structured execution data via display_pub - Include duration, success status, and error type in metadata
- Import and setup execution tracking during runtime init - Add optional execution timeout monitor setup - Configure timeout monitor via environment variables - Add error handling for both features - Maintain backward compatibility (timeout monitor disabled by default)
- Document execution tracking and timeout monitoring - Include configuration examples and environment variables - Provide debugging guide for stuck executions - Explain log formats and locations - Add testing instructions and examples - List all modified/created files - Include future enhancement ideas
- Fix LoggerManager usage: use LoggerManager().get_logger() instead of LoggerManager.get_logger() - Fix webapp URL import: use get_absolute_userpod_api_url() instead of non-existent get_webapp_url() - All imports and functionality tests now pass
- Fix incorrect timestamp: use time.time() instead of duration in metadata - Move time imports to module level in execution_timeout.py - Add threading lock to fix race condition in timeout monitoring - Protect current_execution access with lock - Copy execution data before processing outside lock - All fixes validated and tested
Contributor
|
Caution Review failedFailed to post review comments 📝 WalkthroughWalkthroughThis pull request introduces comprehensive execution monitoring and debugging capabilities for Jupyter notebooks. It adds execution timeout tracking with optional auto-interrupt, execution event logging with metadata publishing, and enhanced debug logging configuration. New modules Sequence Diagram(s)sequenceDiagram
participant Cell as Cell Execution
participant Tracker as ExecutionTracker
participant Logger as Logger
participant Webapp as Webapp
rect rgb(200, 220, 255)
Note over Cell,Webapp: Execution Start
Cell->>Tracker: pre_execute event
Tracker->>Tracker: Record start_time,<br/>cell_preview, id
Tracker->>Logger: Log execution start
end
rect rgb(240, 240, 240)
Note over Cell: Cell runs...
Cell->>Cell: code executes
end
rect rgb(220, 255, 220)
Note over Cell,Webapp: Execution Complete
Cell->>Tracker: post_execute event
Tracker->>Tracker: Compute duration,<br/>success, error_type
Tracker->>Logger: Log execution end
Tracker->>Webapp: publish_execution_metadata
Webapp-->>Tracker: response
Tracker->>Tracker: Clear state
end
sequenceDiagram
participant Cell as Cell Execution
participant Monitor as ExecutionTimeoutMonitor
participant Timer as Timers
participant Webapp as Webapp
rect rgb(200, 220, 255)
Note over Cell,Timer: Execution Start
Cell->>Monitor: on_pre_execute event
Monitor->>Timer: Start warning_timer
Monitor->>Timer: Start timeout_timer<br/>(if enabled)
Monitor->>Monitor: Record start_time,<br/>code_preview
end
par Warning Path
rect rgb(255, 240, 200)
Note over Monitor,Webapp: Warning Threshold Hit
Timer->>Monitor: warning_timer expires
Monitor->>Webapp: POST /execution/warning
Webapp-->>Monitor: success
end
and Timeout Path (if enabled)
rect rgb(255, 200, 200)
Note over Monitor,Webapp: Timeout Threshold Hit
Timer->>Monitor: timeout_timer expires
Monitor->>Webapp: POST /execution/timeout
Webapp-->>Monitor: success
Monitor->>Monitor: Send SIGINT<br/>to process
end
end
rect rgb(220, 255, 220)
Note over Cell,Timer: Execution Completes
Cell->>Monitor: on_post_execute event
Monitor->>Timer: Cancel all timers
Monitor->>Monitor: Clear execution state
end
Pre-merge checks✅ Passed checks (3 passed)
Comment |
|
📦 Python package built successfully!
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## hannes/enhanced-logging #32 +/- ##
===========================================================
- Coverage 72.88% 71.35% -1.54%
===========================================================
Files 93 95 +2
Lines 5142 5307 +165
Branches 754 765 +11
===========================================================
+ Hits 3748 3787 +39
- Misses 1150 1276 +126
Partials 244 244
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
|
🚀 Review App Deployment Started
|
hc2p
commented
Nov 17, 2025
|
|
||
| def on_pre_run_cell(self, info: ExecutionInfo) -> None: | ||
| """Called before running a cell (before pre_execute).""" | ||
| cell_preview = info.raw_cell[:30] if info.raw_cell else "<empty>" |
Author
There was a problem hiding this comment.
I would remove this from the logs or truncate
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds comprehensive execution tracking and optional timeout monitoring for Jupyter notebook cells to help debug stuck executions and improve observability.
This PR builds on #31 (enhanced logging) and adds execution-level tracking on top of the logging infrastructure.
Features
1. Execution Tracking (Always On)
2. Execution Metadata Publishing
3. Timeout Monitoring (Opt-in)
Environment Variables
`DEEPNOTE_ENABLE_EXECUTION_TIMEOUT`
`DEEPNOTE_EXECUTION_WARNING_THRESHOLD`
`DEEPNOTE_EXECUTION_TIMEOUT_THRESHOLD`
`DEEPNOTE_EXECUTION_AUTO_INTERRUPT`
Implementation Details
Thread Safety
Production Safety
Testing
To test execution tracking:
```python
Execute a cell and check logs
print("Hello, world!")
```
Check logs for:
```
EXEC_START | count=1 | cell_id=... | preview=print("Hello, world!")
EXEC_END | count=1 | duration=0.01s | success=True
```
To test timeout monitoring:
```bash
export DEEPNOTE_ENABLE_EXECUTION_TIMEOUT=true
export DEEPNOTE_EXECUTION_WARNING_THRESHOLD=5
deepnote-toolkit server
```
Then execute a long-running cell:
```python
import time
time.sleep(10)
```
After 5 seconds, you should see a `LONG_EXECUTION` warning in the logs.
Files Added
Files Modified
Documentation
See `docs/EXECUTION_TRACKING.md` for:
Checklist
Related
Summary by CodeRabbit
New Features
Documentation